Pictures that are literally worth over a million words

Check out this cool little tool: Google N-Grams

It shows a graphical representation of the frequency words used in books over a rangeof years. It is based on on all the books google has scanned into their database to date.

This TED talk is what turned me on to the project.

The project, the tool and the lecture are all quite entertaining.
Here are some graphs I created playing with the tool. Graphic data, especially that which is based on sentiment represented by our societies authors,  gives us amazing clues into how perception and reality intersect.






































Team City Monitor (with coverage history) for GAE

Per my last post about the splendor of GAE, below is one example of a tool I built for us at Socialize to keep an eye on code coverage and lets the group know if a build is broken (if our tests have failed.) This script is built to run on Google AppEngine, checks against a TeamCity Server install, and expects that each project in team city has code coverage outputted to its artifacts folder.

What is C.I. Server and why is it important?

C.I. stand for Continuous Integration, and is a system used to constantly/”continuously” checks to see if a code base is “working” or not. Team City (by JetBrains) is a product made for that process. Usually it is used to run tests against a code base to make sure it is working as expected, as a part of the QA life cycle. These tests are made up of “unit tests” that are created by the developers whom write code to test the code they are writing. It can be a bit odd for those that have not ever done so, but it is quite important when trying to deliver stable code. In short, and in its simplest form, this whole system makes sure new code/changes doesn’t break old code already in place. You can read more about these processes and purposes here: Unit Testing & Integration Testing.

What is code coverage, and why is it important?

Code coverage examines how much of your code is being tested. For instance, you may have a C.I. system in place, and a unit test frame work running within it, but if the tests only test/”cover” 1% of all of the code you aren’t really delivering a level of confidence you should be. In this tool we track the coverage percentage over time. Seeing it as a graph helps recognize dips in coverage easily.


Download Team City Monitor for GAE


Or fork and contribute on GitHub

Leave the caves and create your tools!

GAE offers a free to get started approach, along with an instant “hello word” initial environment, making sandboxing ideas and building helpful tools for productivuty a snap.

To get started, download GAE, press the plus sign in the bottom left corner. Set the directory you would like to work out of and your almost done. Well, at least you are already at the stage you need to be to start playing with the system locally, in what we call the development environment (No one can see your system but you.) Just hit the [play] button on the GAE dashboard and you are running with your first environment. Just click the Browser (the compass looking thing), or go to http://localhost:8080 in your browser, and you should see your first “hello word! It is quite reassuring to see it work so smoothly (if indeed it does), and if this is the first time you have coded, trust me they have taken out a hell of allot of pain out of the tedium it can take to get here.

Without getting into the nitty gritty of code just yet, let’s push your baby to production (That means make it live/accessible to the world). That’s right, you are about to push a web application live to production!  First create a new app at the google app engine home page and follow the steps there (setting up your yaml for upload). Your yaml file tells google which app your are updating when you do so. Not making sure your yaml matches your project is like  you sending mail through the USPS without out a “from”/”to” address.

Once complete, press the blue arrow pointing upward (the “deploy” button) and it will deploy (AKA: push to prod, go live.)

Once deployed, you can update, monitor, or even share you application with the world. And all for free. Not that this baby would get allot of attention in its current state (just a “hello world”), but if it did, it would also be scalable. I mean 10 years ago this would have cost you quite a bit of time money, especially if you didnt know allot about server configurations, apache, linux, or windows server, or…well you get the idea.

There are a few sample apps you can play with on the GAE site. If you are ready, start developing some code i Python. Maybe had a hellow world message of your own.

When you start feeling saucy, try and create a model. A model is a data structure you can save, or persist, data to your system. Again to you newbies out there, this is the equivelent of your granfather telling you, “back in my day I had to walk up a hill in the snow to get to work, and up a hill in a blizzard to get back.” Setting up a database on a production server was a skill on its own, but to create one that is scalable, and without the need to architect it is amazing. You see, based on the models you create GAE intuitivley creates your “database”, stores it efficiently, and assumes where indexes need to be placed. You really don’t have to understand any of this, but if you want to you can look up those terms have at it: indexing, database, architecture, MVC…. Like I said, I’m just an old guy complaining about hills.

If you are still a bit timid about getting started, don’t worry there is a baby step in between these tween sized steps that can help you get ramped up before you start churning out lines and lines of code. Click on the “SDK Console” button on the GAE dashboard. It will open up a web page that is running locally, on your stage environment, that gives you windows into your system to hack around with. (This console lives inside your development app, so don’t forget to run your new app to get access to it.) Once in the console, click “Interactive Console”. There you will have a very rudimentary terminal that you can write temporary test scripts in. The output is shown on the right of the screen. This is a great place to get errors, make mistakes, and go nuts! (Note: The SDK Console also houses your development DB, so you can check to see what data you are saving after you have attempted to save it.)

Note: The easiest way to get started as a newbie, in my mind, is by using Python in GAE. Java, although awesome, is a bit more advanced.

I recently used GAE to create a few projects to help out the team. One for TeamCity monitor to view coverage reports and if a build is broken or not. And also one for Pivatol Tracker to help our press and marketing interpret what is coming out of the product pipeline, if its ready, and what are the stories of value within. I will post templates for those projects in the near future.

The ol' switcherooo

I realized todaythat with the advents of iPhone and Google voice free calling from my computer I now use my phone for computing and my computer for calling more then the other way around.

Maybe this is anoter indicaton that mobile is more about sponetaity and look ups where the computer is used as a momemt in time reserved for an action. I am walking and curious about the weater i am about to enter, check phone. I need to talk to my mom at 8 sit down and use skype or gchat for 30 mins….

Maybe ist just ironic, but it was a funny thing to notice today either way.

The ol’ switcherooo

I realized todaythat with the advents of iPhone and Google voice free calling from my computer I now use my phone for computing and my computer for calling more then the other way around.

Maybe this is anoter indicaton that mobile is more about sponetaity and look ups where the computer is used as a momemt in time reserved for an action. I am walking and curious about the weater i am about to enter, check phone. I need to talk to my mom at 8 sit down and use skype or gchat for 30 mins….

Maybe ist just ironic, but it was a funny thing to notice today either way.

Google testing out preview pages in search

I stumbled onto the google test list tonight and noticed a new preview feature in search. It works really well and takes good advantage of the excess white space on their search page. A nice bonus is the non forced nature of it. If you mouse over any of the results nothing happens unless you click on one of the magnifying glasses. The full length preview of the page immediately comes up (way quicker than I have ever seen on other search engines) and since its the full page scrolled you can read and browse it quite thoroughly. After the first magnifying glass is selected, subsequent search results show the preview by simply putting the mouse over it. Also if you scroll the preview does not flicker through mouse over events so you can browse the full preview. When you stop scrolling and mouse over a new result it then switches. Overall feel was great and I hope they roll this out.

Google preview
Google preview

Knol: a Unit of Knowladge

Knol
Knol

Knol (http://knol.google.com/) is a product from Google that allows users to post content about whatever they feel they have expertise in to th web. At first glance I wondered if this would cannabalise blogger or any other form of content publishing Google already has on the market. After diving deeper I realized that knol is between a blog and a wiki. In that it’s not just for posting information in a time line like a journal, although there are plugins to make blogging apps work less like a journal and more like a site, and it isn’t modertated bu some elite “truth finder” entity like a wiki.

About 6 months a go a friend asked mine asked me for some direction when trying to figure out a way to put up what he knew about his local citities night life, as he found many people came to him for suhc data, but he didn’t want it to neessacarily be a blog. A blog in his eyes was to date based and abstract. He wanted a different interaction model that allowed users to dive into specific ascpets of information about going out in his city. Not knowing of Knol at the time I told him the simplest thing to do would be to start writing a blog and if there is traction but some extra time into manipluate the front end to cater more to his needs through stylings and graphics or plugins. So based on that interaction with him then I can see exactly how Know could have an audience to cater to.  I offered this tool to him recently and he seemed pretty excited about it. I awat his response on how much better it is then the blog and I’ll be sure to let y’all know what he finds as soon as I get the info myself 🙂

Google Voice: A look around

Google Voice
Google Voice

Finnaly got the invite tonight from Google Voice, Googles completly free phone forwarding and voice mail service. Google voice is the new version of what was fromerly known as grand central. The servcie lets you choose a phone number and all calls made to that number are forwarded to a number of your choice, be it cell phone, home phone or work. There are a good deal of options and setting that come with this service and overall it seems to be a pretty cool evolution in phone messaging. Phone VM transcription services and magaement systems have been around for years but as usual Google has made this service free now so other small businesses will be hard pressed to beat that bottom line.

GV Phone App
GV Phone App

One great feature is the transcription service for all voice mails. When a user calls and leaves a voicemail message Google takes the voice stream and converts it to text. You can then read the voice mail via email, text message, through the Google Voice portal or through the downloaded phone application for Android or Blackberry. The transcription works well pretty well most of the time, although it’s not perfect just yet. Phone numbers were deciphered 100% accuratly, so far at least, and you can always play the voice mail along with the text if you have any problems with the transcription. Also, on the phone app the text is higlighted as the voicemail is read to you.  If you want multiple numbers going to and from a single number check out vumber.com for multinumber. Note: There is one place that Google Voice charges, international calls.

Phone App VM Inbox
Phone App VM Inbox

As far as anyother charges that one could inccur from the phone network, I called T-Mobile and the representative said that many companies in the VM business have worked payments sytems with phone carriers out to make sure although you aren’t chareged for these services you dont abuse the carriers lines either. Of course rates will vary between plans and carriers so make sure you check your plan for details but in my plan I get 500 forwarding minutes during peak time. If  thise minutes get used up then forwarding gets counted against my regular minutes, once again only during peak hours.  All in all it would take alot of long messsages to go over those limits so I am happy with both sides of the system so far. I will see at the end of this month if what I was told ends up what i see on my bill..

Also, Daniel tunred me on to some cool settings for GSM phones. You can force your carrier to use a third party VM service so all your calls either through Google Voice or through your own personal exsisting number uses the Google Voice VM service. You can find out how to set that up here (http://go.danielodio.com/voice). Bassically dial *004*1[yourVMservicenumber]# (i.e. *004*15551234#) and all future VM calls will be sent to the VM service phone number provided.

Below are some screen shots of the dash board and settings provided by the Google Voice service.

Dashboard inbox diplsay VM messags and their transcriptions. You can send SMS messages and call from your dashboard as well. If you use gmail all your contacts come along with the dashboard so who is calling is not just a phone number but a name and picture could be associated with each entry.

Home Screen
Home Screen

This cool feature below lets you create VM greetings that change depending on who is calling you. Have a professional greeting for unkown and work contact and a fun one for friends.

Settings 2
Group Specific Setting

Send transcriptions to email or via SMS, screen callers or keep calls from ringing your phoen and going straighth to VM with do not disturb option.

Settings 1
General Settings

Google also has a widget that lets you put a contact me area on yor page so people can call yo without having to know your direct number.

Settings 3
Call me widget

Quantify my language

oDesk Search Results
oDesk Search Results

The cost to scale a web application has changed considerably and therefor the way most people decide to plan out thier development strategies has changed as well. Not every web site is an enterprise solution and getting many sites or versions of sites out to the public quickly has become the greatest focus on a fledging comapanys mind.  Because scaling is so cheap and servers are so easy to set up with clouds I often hear, ‘lets deploy first and see what people think and if it gets big we will just through some servers at the problem, and improve the code as we grow.’  For the most part I agree, in the case that a complex problem arises an RPC call to some seperate more efficient code base, server or algorithm can be used.

Dividing up a site into 3 parts is key, re-inventing the wheel tasks, already built better then we can build it widget/api tasks, lets solve a new problem tasks. For example: login, and register has been developed 100 times over as well as setting up projects and DB connection settings. Getting a users feedback or publishing your content to all the social networks has been done beautifully by many companies that now give it away for almost nothing and are great to use, unless your new great idea has a core innocation in such technolgies. Then there are the problems out there that may be partially solved, but there is still quite a bit of technique and brain power needed to tackle your unique version of the problem to complete your companies mission, like improved search of billions of peices of data or 3 degree+ of sepration colculations. The mission is use what you can that already exists if its not your CORE business to own such IP (often times people think they have 5 core’s….they don’t) and use what you can to get that core going as quickly as possible. This isnt true for all web startup’s — but it is for most.

oDesk Feedbacks
oDesk Feedbacks

I will save more on that topic for a later post. Today I simply wanted to take the problem of getting started up a level or two and look at the first part of development for non developers.  I mean I personally know what I like to develop in, and it’s always good to ask someone with more experience then you what he/she thinks, but I all to often I find people that have inadvertenntly chosen the more expensive, more timley path and in the end they have a mess, a bill, and an almost incomplete site. They are then left with finding a batch of new developers who can jump right in and take over. I see this ALLOT and it breaks my heart. Is it jus a product of bad consulting? Is there a way to help get non developers a leg up and allow them to make good desicions quinatitatilbyl without a consultant telling them biasedly which way they can go to suit there own needs.

I know that oDesk, and other internet based freelance servcie is where allot of folks turn to so I went out and got some basic datapoint to start this conversation. Below are some charts I put togther using search scripts to get results sets from oDesk (whom I think should put out more repots on this matter to help out the public).  I simply tried to query things people may be wondering and get results sets for each lanuage respectivley. I was suprised to see how flat the results were.

oDesk Search Results:

# high rated   feedback (2+ FB & 4+ stars) total   feedbacks (4+ star) most hours (2+ FB & 4+ stars) $/hr of most hours (2+ FB & 4+ stars) avg top 10 hours avg $/hr of avg top 10 most hours avg cheapst top 5 avg most exensive top 5 highest reqested rate
Python

3

6

606

$33.00

146

$25.34

$24.67

$28.10

$83.00

Ruby

12

9

2300

$16.00

663.5

$27.50

$14.93

$95.51

$230.00

PHP

64

37

4045

$30.00

1923.7

$26.44

$9.33

$75.62

$166.00

Java

12

15

712

$27.78

530.3

$26.13

$8.45

$54.09

$166.00

.net

10

14

2490

$19.44

641.6

$26.85

$11.55

$40.80

$111.00

Below is a list of some languages and how many hits came up when the languages marked with an X where combined in the search. I guess what I was going for here is how many people know what combinations of languages. For example people who posted a knowlagdge of all languages was 7, and those that claim to provide services to both python and ruby were 35.

Java

Python

Ruby

.net

PHP

Search       Hits

x

1427

x

246

x

300

x

1757

x

2397

Expand to show all results +

oDesk Results
oDesk Results

I will admit I dont like .net and I am not a fan of PHP as much as I used to be and although I enjoyed building FamilyOven.com in Java iterating over quick changes on the fly got really annoying having to compile and recompile my code. The pside of course is its great stability and easy to debug format. It’s just that scripted code suits my needs more these days. I put things together to push out new sites and stabiiy in the first year isnt that hard to do with server costs the way they are these days.  Python and ruby have allot of abstraction but for the first year I probably wont care, second year I will just pull hard tasks out like I explained in the first part of this email.

The data I gathered made me seriously wonder for a moment, am I just falling into the hype or is python and ruby as smooth, efficient and easy to develop sites quickly as great as it seems? I didnt jump on the ruby, python band wagon quickly. I created a few sites on it before I formed an opnion for sure.  I guess whats more important to notice here is how the hell is a non-technology/development person able is to decide on a languge to go with after talking to vendors and seeing results like these. I mean these data points love .net and php as giving off the impression that they are affordable, and popular.  I feel however that those that I know that use Ruby and Python are more innovative, fast, and love what they do more so then the others, not to mention cheaper in that they get more done more quickly. (or so it seems that way as of late) I would love to get some advice on ways to mine for data that respond with data that can statistically prove my sentiment isn’t just an opinion, a sort of freakenomics style of analysis if possible. I dont want to try to manipulate data to prove my point I just want to see if there are any true data points out there that confirm what my gut tells me.

Of course being that ruby and python are new compared to the other languages one could expect such unimpresive numbers and stats, I also think that being that they are so new to the common developer tool set it is impresive just how many people code on it already and how many plugins are available. One metric that could be usefull is expense to get started materials wise, free easy to use documentation, number of plugins, and time to start a basic project/site.

Python for instance is part of the Google standard languages, that speaks highly to me. It is supported by the Google App Engine and it has jython a frameowkr taht allows python to run in java runtime. That is mportant as well as java run time is more efficient then most things out there so you have flexability. Ruby has tons of plugins as well as python. and all scripting languages take almost no time to set up.  Also it is important to pointout that other then biz spark from MS getting a project developed is MS systems are not free to get started on from servers, to dev environments to documentation and support many MS product cost quite a bit which can get in the way of trial and error, innovation, speed to market, and keeping initial expenses down.

Please comment if you think I should account for a factor I missed or if youhave some data on your own.  If you have some crutial data points please list them and the source you were able to get it from. THe more straight data we get the more we can help people make better descisions rather then just using he told me so approach.

Here is some data about the job market in 2008. More data on jobs and how they relate to development would be helpful as well. i.e.  http://dotnetslackers.com/Community/blogs/xun/archive/2008/08/18/who-has-respect-in-web-development.aspx

Pie for mass consumption

Advanced Pie
Advanced Pie

Yet another great Google API for the masses, on the fly charts. You can simply use a URL and set of parameters to describe your chart and the output is an image file.  So, use the url in your blog post or place it in your href tage and get a great looking chart in a snap.

Down side, there arent as many options as I would like through the API URL; I cant show perentages on te pie chart. Put they look great, they are easy to use and you can update them on te fly simply by changing the URL.

Check out more information on this cool little feature at http://code.google.com/apis/chart/

or try one of these examples:  Pie Chart, Bar Chart, and even a QR Code.