Small code with powerful results, the occasional opinion … and beer. 

10 Nov 2014
The Third Man to credit for the fall of The Berlin Wall

The Fall of the Berlin Wall is an event that is very personal to me, since I was stationed at Tempelhof airbase in Berlin with the US Air Force when it happened.  As the 25th anniversary of the opening of the Berlin Wall is being celebrated in Germany, there are two key people being credited and remembered for the opening of the East German borders that occurred on that November 9th, 1989. One was the [East] German Democratic Republic (GDR) politician who mistakenly mentioned that the borders were being opened, and the other was the GDR border guard who ultimately chose to open the border due to an unclear situation with his leadership. The NPR article referenced here has the details of who they are, and how this played out:

The Man Who Disobeyed His Boss And Opened The Berlin Wall

There is another person who deserves some credit for the ultimate opening of the border, although his actions occurred weeks before Nov 9th.  Still, his critical decision opened the door in East Germany for the wave of democratic reforms, started by Gorachev with his Glasnost policy in the former USSR, to culminate in the events of November 9th which truly ended the Cold War.

That person is Egon Krenz, who was the General Secretary of the German Democratic Republic on that day.

But to understand his contribution, it is necessary to first go back to May of 1989. It was the time when the protests for democratic reform were occurring on the other side of the world in Tienanmen Square, Beijing China. The ultimate result of those protests were a harsh crackdown by the Chinese military to disperse the crowds and arrest the leaders. It resulted in a lot of condemnation of the action by the world, and also left the indelible impression of “tank man”: the lone unknown man who stood in front of a set of tanks trying to leave Tienanmen Square and brought them to a stand still. The symbolism of that one act was a catalyst for the wave of changes that swept Eastern Europe in the coming months.

Fast forward to October 9th, 1989, to the town of Leipzig, in the southern part of the GDR.  At the time, Egon Krenz is not yet the General Secretary of the GDR.  Erich Honecker was still the General Secretary of the country, as he had been since 1971. Der Spiegel has a great article on how this huge protest started. It was a major step to the Berlin Wall coming down.

‘We Are the People’: A Peaceful Revolution in Leipzig

This large, spontaneous protest put the GDR goverment under similar pressure as the Tienanmen Square protests had done to the Peoples Republic of China.  How does the government react to the protests? Ultimately, General Secretary Eric Honecker made the decision to deploy the military to suppress the protests. He had been a big advocate for following China’s example of dealing with the reform movements. Egon Krenz received the order for Honicker and, for whatever reason, ordered his subordinates to ignore it and made sure that it would not be implemented. His decision undoubtedly saved numerous lives.

Not long after, an unpopular Honecker left his office, and Egon Krenz replaced him as General Secretary of the GDR. He held that office during the border openings, and until December of 1989. Interestingly, he became one of a number of former GDR officials in the 1990’s who were tried and imprisoned for a period of time for the human rights abuses related to the killing of people trying to flee the country.

Krenz’ contribution to an almost completely peaceful fall of communism in East Germany is not that well known outside of Germany and deserves mention. You can read more about him specifically in the article below. He is commonly labeled as a Communist apparatchik, was only at the pinnacle of his career for a very short period of time, and probably spent time in prison symbolically for other people who should have (but no longer couldn’t). Yet, he was definitely in the right place at the right time, and made a very good decision that prevented would would have been inevitable bloodshed.

Hero or Villian? Egon Krenz, Communist who got to the top just as the Party was over.

31 Jul 2014
The RFID implanting scare, and an incorrect comparison in the Snopes.com analysis.

** Before implying anything about Snopes.com from the title, please note that I refer to Snopes.com often and find their information to be quite useful for a quick comparison of what I am reading against known facts. They are not the final source of information for me, but have proven to always be a good starting point.

This article notes a very specific inaccuracy, and I only publish it here to emphasize the importance of criticial thinking with every source of information. I think anyone running a site like snopes.com, which undoubtedly deals with a near flood of rumors to investigate and document, would have potentially overlooked what I am documenting here. **

Back in July of 2013, snopes.com posted an article about an end-times email scare.  The scare was a strange mix of misinformation about the health care law, combined with claims about radio-frequency identification (RFID) implanted chips mentioned in a broadcast NBC report.  As the snopes.com article correctly shows, those claims weren’t really there in the NBC report, nor was there any such language in the health care law.

The original article is here. I only recently found the article, because the email finally made its rounds into my inbox.

So what’s inaccurate in the snopes.com article? The snopes.com article contains a snapshot of the original photo of the RFID chip which appeared in the email mentioned (image to the right below), and also has a link to the product page which Snope’s claims is the RFID chip in the email (image to the left). I’ve put them side-by-side below for comparison.


Snopes (from medgadget.com)

Emailed

The snopes.com article claims the chip pictured in the email is the same as in this article.  The exact quote from snopes.com is the following:

Some May 2012 versions of the hoax circulated by e-mail and Facebook postings displayed a photograph of the purported chip meant to be implanted, an item described as the size of “a grain of rice.” The chip shown in the photo is actually one that measures glucose levels in diabetes patients, as evidenced by this 2007 article about this new concept in glucose monitoring.

I disagree with Snopes’ conclusion for the following reasons:

#1: The person in the image isn’t simply altered with photoshop to blur out the face in one picture.  There are a lot of things different between the two people.

  1. The position of the finger above the eye is blocking two different portions of the eye in each picture.
  2. The photo in the email cannot be a cropped version of the original photo in the medgadget.com article because you have to have something to crop.  Notice that the view in the email photo from left to right shows more facial area than the photo in medgadget.com, which it is supposedly copied from.  Cropping shows a smaller area of a larger photo, not the other way around.  There is no nose in the medgadget.com photo to copy.
  3. The fingers in the email photo show skin and fingernails: in other words, it is a bare hand.  The fingers in the medgadget.com photo are wearing latex gloves.

#2: The chips in the pictures are clearly different.  Note the following differences.

  1.  The medgadget.com photo shows brass-colored sensors at the end of the chip (bottom).  The email photo does not have them; it is a solid clear shell.  This makes sense since a glucose measuring chip needs sensors on the outside to make contact with the fluids to measure them.  An RFID chip does not need external sensors: it is not measuring anything.
  2. The medgadget.com photo shows a thicker shell on the top half of the chip, while the the email photo does not have this.

The chip in the email is a Verichip product, based on a Destron device available since the 90’s. The product slip for it is here. This chip has extensive use in research laboratories for identifying animals in experiments, and is a major advance for this industry. Prior to implantable RF identification, mice chewing off an ear with a tag could ruin days or weeks of research efforts if the animal could not be identified. It also allowed the identification number to be read directly into the notebook/laptop by waving a wand over the animal (no danger of invalid number entry from a human interface like a keyboard).

The device pictured in the email is similar to the device at the top left on the product slip: there are no external sensors on an identification responder. Functionally, the chip in the email and in this product slip is similar to the automated highway toll paying device in your car. When a transmitter at the top of a toll-booth gets close enough to the device in your car when you pass under it, the device in your car transmits its identification number in response. That is all it does.

Any other conclusions about the purpose of the device in humans, whether it is the mark of the beast, etc, are up to you. I am merely pointing out that the chip in the email is not the chip which Snopes.com claims it is.

(original NBC video mentioned in the email)

22 Jan 2014
Big Brother at its best? Well, maybe…

Recently, it was revealed that the NSA is able to break common encryption protocols used on the internet.  You can read the article here:

http://www.theverge.com/2013/9/5/4698808/most-common-encryption-protocols-are-useless-against-nsa-surveillance

But, to me, it’s not really that much of a revelation.  And taking the article’s claims at face value will cause the user to miss a practical side of security that is often overlooked.

Having been in the technology world for close to 40 years, I read a lot of material related to technology.  Around the late 1980’s when PC’s still dominated the computer networks and the internet was still not opened yet for general public use, there was an article about 40-bit encryption and even 64-bit encryption in the marketplace.  It also mentioned that the academic community was already talking about 128-bit encryption, but that there were signs that the NSA was not going to allow this.

Why would the NSA not allow it?  Because quite frankly, if the NSA is not able to decode it, and the cipher gets into the hands of people with evil intent, it can’t be monitored for abuse and it can be used in many evil ways.  One of NSA’s tasks is to ensure the security of government communications.  This task falls into that realm.  If there is no foreseeable way to break the code, you cannot deny its use to people who would use it against you.  This is one of the reasons why the United States has laws banning export of certain encryption technology.

That’s why I am not surprised that NSA would have the capability to break the encryption we use on the internet. BUT DON’T MAKE THE MISTAKE OF THINKING THIS: The NSA can decode everything I do on the internet now and read it.  It’s easy to come to that conclusion because of the stated ability to break a cipher.  But there is another aspect to breaking a cipher key that is just as important: how long does it take to actually break it.

Did the article mention anything about how long it takes?  Of course not.  That’s the real secret, right?  Even though NSA has the ability to crack the keys, the actual time spent on the effort could be days, weeks, or months of time and involve thousands of CPUs running in concurrence to solve it.  Once the key has been cracked, all the data collected to that point which used the key is now decipherable, but not before then.

In any security system, there are two core aspects to it.  The first is the physical or virtual layers of security intended to stop access to the item being protected, to prevent unintended use.  In this case, that is the data.  The second, is the time required to gain access.  How long is the lock I installed going to stop a criminal from finding the key, and making his own copy to gain access?  In a practical example of a break-in, a thief may be trying to gain access.  If I add an alarm to the front door, even if the thief has figured out the key to get in, how long does he have before he is discovered and arrested by the police responding to the alarm.

Of course, if the thief has found a way to get access to your key chain, the time he needs to spend picking the lock mechanism to get the right combination of tumbler settings is zero.  So if NSA has found a way to get at the cryptographic certificates on a server or network device directly (the virtual world’s keychain), their time figuring out the cipher values also goes to zero.  But I doubt this capability exists, short of a software bug or forcing some sort of malware onto the system to help them into the machine’s pockets.  This latter attack is what modern day virus scanners look for.

So just because the NSA may have the ability to break an encryption key doesn’t automatically make that a bad thing. Quite frankly, it’s important to remember that our everyday activities online draw more interest from commercial business activities than government security interests.  The real concern we have is not with NSA, which is just a tool for acquiring and analyzing the data.

The real issue of privacy goes to motives of the people who keep pressing NSA for information and try to pass laws to circumvent constitutional rights to get it.  That is the real problem.

 

 

 

 

 

 

 

05 Jan 2014
User beware: the hidden gotchas of the Verizon FIOS service

Verizon FIOS is a fiber-optic system capable of delivering very fast internet feeds to residential consumers, in addition to its TV service. With upload speeds in excess of 25Mb, and download speeds in excess of 50Mb (its currently capable of 150Mb), you would think that this service beats cable internet and DSL internet hands down.

Well, speed-wise it does.  But there are quite a few gotchas, including some not so apparent security risks. If you are not a casual residential user of the service and take the time to login to the router , you’ll quickly discover the Actiontec router provided by Verizon is a flashy, poorly designed child’s toy.  While the hardware is solid, the choice of firmware is dismal and, in my opinion, more than a bit dangerous.  It looks and behaves like someone’s abandoned science project, which was picked up and finished by the marketing department at Verizon.  There is no common sense whatsoever in its design or user-experience.

And worse, the firmware also initiates strange connections to Verizon servers, which make me question the router’s security and integrity.  Since I am engineer, I find it appalling what Verizon gives out as a core piece of the network in a user’s home–especially in light of the recently-revealed NSA eavesdropping and network penetration efforts and, before that, the years of black-ops efforts on the net to seize control of networks for bot armies, industrial espionage, monetary theft, etc, etc.

The real test of a good internet connection is not only the speed and how much your network can do for you, but more importantly how much people outside of your home network don’t have a chance to compromise it and, even worse, take control of it. Verizon FiOS architecture fails, quite frankly, very miserably on both of these accounts. Here’s a specific set of reasons why, broken down by the level of importance.

Security and Network Ownership/Management

The Actiontec router has two network interfaces for the WAN (Internet-facing) connection. One is coax, and one is ethernet.  The box is setup by the installer with the coax connection, because the box is designed to work with the DVR unit to access the network for TV program information, etc. And, surprise surprise, the DVR only has a coax connection to access the internet.

This is a very subtle, and very dirty trick to dissuade users from disconnecting the Actiontec router and putting in their own router. Publicly-available routers are known to use standard Cat 5 network connections, so a standard router won’t directly support a connection to the DVR. So this quagmire of giving up your onscreen programming guide to use your own router is created.  Most average users will give-in to using Verizon’s router because they don’t want to give up the programming guide on the TV, and don’t have the knowledge of how to work around that with their own router.

Verizon, for any number of reasons, would love to control the traffic on their network–even to the extent of managing their company assigned router inside your house to enforce their corporate policy and thinking.  This is a very dangerous way of looking at the internet, which is designed for a free-flow of information.  I have documented a legal move made by Verizon in the past in this related post here, written a few months back, which demonstrates why this is their motive.

In addition to the business trick of discouraging alternate router usage, there are also some additional, open ports on the Actiontec router which indicate that it is/can be centrally managed.  Centrally managed means the router can be exposing its settings. logs or even receive remote firmware upgrades at the will of Verizon.  This would violate the cardinal rule I have for any piece of electronic equipment which I own: updates allowed only when I am notified and approve.

While some people will argue that this means a security hole can be patched quickly across the network, the converse is also true.  Because a large set of routers are available to a central management system, an intruder with ill-intent could potentially put a compromised firmware into that system for distribution.  Less aggressively, a release of firmware which has an undiscovered problem could potentially take thousands, if not millions of households offline at one time.

And worse, because Verizon is a publicly traded company, the problem could be concealed, or described in a more generic form as a “network issue we are working to resolve” to mask the real cause of the problem in an attempt to protect stock values.  In the open source world, which DD-WRT is a part of, many people contribute, test, openly write about and scrutinize the software.  Because of the openness, the user has enough information to decide if an upgrade to their router is appropriate.  And if they decide an upgrade is appropriate, they decide upon the time.

Even if DD-WRT were compromised, the chances of it being discovered and exposed are far greater due to its very open, public nature.  Not so with Verizon’s approach.

The Awful User Experience of the Actiontec Router’s web management interface.

In some ways, there are too many pain points in this browser interface to list.  But I will list the ones that stand out to me.

  • Trying to get the user lost the moment they attempt to login.  The very first one starts with the login screen for the router management.  As keystrokes are entered into the password text box of the dialog, the router will actually change the number of asterisks that appear to a larger or smaller number than actually typed.  This is so dumb.  Not only does it confuse the person who might be looking over the operator’s shoulder (the intent), but it royally confuses the operator as well.  When the feedback of what is being typed is not displayed, the only measure of accuracy the person typing has is cadence–a count that can match where the operator expects to be in the sequence.  And Verizon’s interface even screws that up.  I can not emphasize how asinine this is.  Most modern username/password dialogs today have an option to unmask (i.e. don’t hide) the password.  After all, if you’re the only one in the room, what’s there to protect?
  •  Locking yourself out of your own router.  Want to have fun?  Enter a bad password in the password text box, and click the login button several times.  The box will actually lock you out, of your own network in your own house.  Every other router on the market will give you infinite chances to login to the router, if you are connecting from something that originates in the house (Wireless or LAN connections).  It is only the WAN origination points (somewhere from the outside to the network in the house) where a certain amount of consecutive failures will cause a lockout to occur. I was just stunned when I saw this.  Make me get up and recycle the power on my own router to try again, because you (Verizon) threw off my cadence when entering the password–come on !
  • Extraordinarily poor navigation.  The items are all over the place, poorly grouped, inconsistent, and diving down to a menu item often requires you to go back to the top and navigate all the way back down again for another action in that same area.
  • No attempt to memorize any recurring answer the user gave.  Certain areas are labeled as for advanced users only, requiring a click-through to approve going into them.  But each successive time you go into another “advanced users” area, you get asked again.  Add this in to the continuous deep-dives needed in the entire menu system, and the amount of time wasted for simple activity is astonishing.
  • Advertising right on the home page.  This is the most laughable to me.  Once you login, and every time you cycle back to the home page (which it does force you to do a lot), Verizon’s router displays advertising links on the right panel–of an equipment configuration page on the local router!  For those of you who wrote this site and let Verizon make this a requirement of you, super glue a brown paper bag of shame over your head.
  • A mysterious port which you can not disable.   The router has NAT, but has a port authoriztion (TCP 4567) that is untouchable by the user.  This should be an automatic red flag that something is going on with an outside server, which Verizon will not allow you to turn off.  The port is known to be a point-of-access for Verizon to enter the router for their purposes.  They will call it customer support, but both the Actiontec and Westell boxes have been attacked and compromised on these ports. http://forums.verizon.com/t5/FiOS-Internet/Guy-accessed-remote-administration-port-4567-on-my-router-Thanks/td-p/241017

Despite all of this, I still have Verizon FIOS as my ISP provider.  As long as my router is the main entry point to the home network, I can manage and protect it as I need.  I do find the path that Verizon has taken with this architecture very concerning.  It would also not be completely fair to say that Verizon is definitely the only one doing this, but be aware of the implications of using the company provided equipment for your home network.

 

01 Dec 2013
DD-WRT: Making two-routers work on the same network… and an alternate

I recently moved and had to setup a new home network. Because of the length and number of walls limiting effective radio coverage in the new house, I opted to put in two wireless AP’s: one at the front in the living room where the main input is, and one at the back of the house (about 70 feet away). I needed to connect the remote router to the network, on the same subnet.  Both of the devices use DD-WRT.  The type of network I am decribing is documented on the DD-WRT wiki here.

The DD-WRT forums had a number of discussions about the difficulty various people have had doing this.  I use two Rosewill RNX-N300RT routers with DD-WRT v24-sp2 (03/25/13) std – build 21061 installed.  In the network, one is connected to the WAN (the ISP access).  This is the primary router, where all the NAT occurs.  It operates it’s wireless LAN (WLAN) with a unique SSID on a specific channel.  The address of this router is 192.168.1.1

The secondary router has no WAN connection, and it also operates its WLAN with a unique SSID on a specific channel.  Both wireless access points use WPA2 for security.  Connecting the two routers is a straight Cat5e cable running under the floorboards for a full 75-feet, connected to one of the four LAN ports on each end.  The address of this router is 192.168.1.2, set as a static address in the secondary router, with the gateway set to 192.168.1.1 to forward all non-subnet traffic to the primary router.

This setup has been quite successful.  The trick to this is to simply setup the second router as described in this document, and most importantly, use the version of the software above  or a later.  The previous build of DD-WRT I was using (19xxx) would not work properly in the secondary router no matter what I did.  Build 21061 worked perfectly, when following the setup described on the DD-WRT Wiki page above.

AND AN ALTERNATE…

Over the Thanksgiving holiday, I visited with some friends who were having similar issues, but the option of running a LAN cable to extend the network was not an option.  The network suffers from the same issue of trying to go through several walls over a 70+ foot distance, but the router and cable modem are located over a spot of earth, which blocks access to the basement.

For this, I opted to go with a Linksys RE2000 network repeater.  I followed some advice to use the same manfacturer for the repeater as the existing router.  The network extender was easy to setup with the software provided, and it was placed in the corner of the living room.  That location is in the center of the house, and past the edge of the basement wall which enables radio-wave passage to the basement.

Result: the wireless is now accessible from the entire house.  And the signal from the extender is strong enough to be picked up just outside the house on the patio.  This is an option I recommend, if you are not interested in stringing wires to extend your network.  Just be sure to match the manufacturer of the network extender to the manufacturer of the router.

:-)

20 Sep 2013
Of Patents and Trolls…

Isaac Asimov, the famous science-fiction writer, died not long ago.  In the obituary I read about him, there was one interesting statement he was quoted as saying during his life.  One day, while apparently resting in his yard, he came up with the concept of bouncing a radio signal of a man-made object in space to send that signal to the other side of the Earth.  Asimov was quoted as later saying, “If I had written that idea down and filed a patent, I would have been a billionaire.”  His concept, of course, was that of a orbiting communications satellite–something commonplace today, but didn’t exist back at that time.

Isaac Asimov stated the exact problem with Patent Trolling occurring against US technology companies today.  The patent laws in the United States do not protect something as simple a writing an idea down and saying you invented it (a little more, in fact, that Asimov said he did).  The courts have ruled on that many times.  What a patent law is designed to protect is the inventor’s investment in the development and marketing of the idea, to bring that idea into fruition as a viable product or service for consumption.

I love hearing business seminars where the speaker will joke about a product idea they once had: usually it is mint-scented toilet paper.  While it is amusing, I say it is safe to assume that thousands of people have had that product idea at some point.  Imagine if one of them took the time to patent it, then develop, manufacture and market that product.  Then imagine what would happen if, to everyone’s surprise, the product took off and became a big hit.

At that point, other companies jump on the bandwagon and begin manufacturing and selling mint-scented toilet paper. It’s now a proven market, and a quick buck. But the patent-owner now has the right to legally approach these people producing the same product with cease-and-desist orders, or negotiate a license with the other manufacturers. This latter agreement would allow them to continue, but ensures the inventors are financially rewarded for their development efforts.

By granting an exclusive right to the invention via a patent, the inventor is protected from people attempting to shortcut the development cycle, and simply manufacture and market a product or service they did not invent–with no concern for the original inventors efforts to bring those ideas to fruition.  It ensures that the inventor reaps the rewards of not only having come up with the idea, but manifesting the idea into a viable product or service that is marketable.

Patent  Trolling results from an idea that stops after the patent is issued.  The troll then goes after the product or service being developed and marketed, having done virtually nothing for development or other steps necessary to manifest the idea. This completely inverts the scenarios above.  Instead of the law protecting the inventor and efforts to develop, manufacture and take to market, the law now “supposedly” protects the patent trolls ideas they only wrote down.  The troll is really doing nothing more than setting a trap to steal the inventor’s harvest.

This is so wrong and harmful to promoting industries that require heavy time and investment–especially software development.  Most times, the trolls are hoping not to have to go to court.  They sue for just enough to make it unprofitable for the company they target to fight the case, instead of settle.  Even though the Trolls may lose their battle in court, the amount of financial bleeding that occurs for the company trying to defend it could be devastating.

While the US Congress has been holding hearings on the problem of Patent Trolling, one US tech company has the best approach I have seen yet.  They counter-sued the Patent Trolls under the Federal RICO Act: the anti-racketeering law which has been so useful in prosecuting mafia and gang members, where other laws have failed.

Whether or not the company wins the suit is uncertain, but at a minimum it should send a clear message about how the activity of Patent Trolling should be seen: as organized crime.  The link to this article is below.

http://arstechnica.com/tech-policy/2013/09/angry-entrepreneur-replies-to-patent-troll-with-racketeering-lawsuit/

 

 

30 Aug 2013
The 5 Core System Requirements To Protect Your Sleep

I have been a software developer for over 30 years now.  That includes time as a kid playing with technology, time as a student, and professional and non-professional work.  Over that time I have encountered a lot of bad designs, and some good designs in systems and software.  I also have, like every person in the engineering or operations side of technology companies, lost a lot of sleep from time to time, due to midnight emergencies or insecurity about having a stable system during peak usage.

I’ve distilled my overall experience into a list of what I consider the fundamental requirements of a system (enterprise or, more simply, multi-server) to

  1. Ensure it is survivable (more than just the original developer can make it run)
  2. Keeps its ability to evolve over time
  3. Minimizes the amount of technical debt created by its design
  4. Minimizes the amount of disjointed interaction, and maximizes discovery and effectiveness of community effort.

I am focusing here on the big picture: the system as a whole.  Best practices for code styles, library use, etc are only implementations of these suggestions which are up to you as a business to decide.  These requirements are all agnostic to a particular platform or technology.

#1: Keep configurations, to the fullest extent possible, out of local .config/.xml/.ini/.cfg files, registries, and other application/service/server-specific locations.  Centralize it, with a good user interface that configuration specialists can use to review and adjust everything.  

This avoids the nightmare of relocating hardware or services, and potentially not having the entire set of configurations needing changes due to the migration.  There is nothing worse than a series of inconsistent tools, services and applications written by several different developers over time, each having their own philosophy or practice of how to store configurations for various items.

Enforce this requirement: it will give your system a chance at longevity and retain scalability, and drastically reduce technical debt.

#2: Log what your application is doing, not only to a specified level of detail set in a configuration, but log it to the RIGHT location.  The RIGHT location has broad centralized access with search and filtering abilities.

This avoids the nightmare of not being able to properly troubleshoot a problem, because the details of an error or step in a process are not recorded.  Personally, I am a strong advocate of chatty log entries.  In my experience, deep-detail logging is useful for the most recent 96 to 120 hours of operation.  After that time, the log entries can be pruned down to a summary level for process accounting.

By keeping the details for 96 to 120 hours, someone can return from a long weekend and have enough information to troubleshoot some problem occurring over the weekend.  The most important details to log are information about a specific step being done, error trace and stack info, and breadcrumb information (specific file names, URI or other resources).  Logging should also provide some information about the server itself (at a minimum it’s system name), to ensure that the reader is aware of where the process occurred.

If the logging system is well-defined and well-exposed, it becomes an excellent foundation for others systems that will be of great value: metrics and alerting.

#3: Have a way to uniquely identify your application or component, within the collection of applications and services on a server, and also within a sea of servers in an infrastructure.

This partially applies to logging, and partially to configuration.  It is also intended to apply to contracts, permissions, service enabling, etc, which are related to the business side of things.  Unix and Linux engineers love deep paths or dotted identifiers.  While it does have to be this technique, you get the idea.

#4: Have a place to validate a business relationship for a customer consuming the process.

As software engineers and system designers, one of the critical considerations is the ability to allow the business administration to control what is available to a client.  If accounting has not been able to get a client to pay, how is the service for that client disabled?  If the system does not have an inherent ability to check this, outside of system configurations it uses for operation of the system itself, then some form of “workaround” develops that may be more for technically disabling the service, then denying access to it.

It is not uncommon to see a service for a client disabled (as part of a bulk action) in order for technical maintenance to be done, then that service is mistakenly enabled for the client even though there was a business reason not to.  If there are not separate switches designed for technical disable versus business disable, it’s easy to cross the streams and get confused.

#5: Implement Flight Tracking

While logging tells you what a process has done (right or wrong), it will not tell you whether a process either ran when it should not, did not run when it should, or … just stopped processing.  While the latter (stopped processing) can be determined from a log, it must be discovered.

Flight Tracking is a concept borrowed from the aviation industry.  When a pilot plans to fly his aircraft from point A to point B, he files a plan with his intended departure time, his intended route, and his destination.  The pilot can cancel the flight plan if needed before departure, or he can make changes as needed.  But the flight plan’s purpose is to know that the pilot and his place is where he said he would be, and react if the aircraft is overdue and out of communication for a period of time, or did not even depart as scheduled.

This is a good practice in an enterprise system.  A specific process should report its launch to a central location, and send periodic updates that it is still running and processing.  Ultimately, it should report its completion.  By doing this, a layer of monitoring can be added to the flight status compared against the flight plan, to report processes which have not provided updates (hung or crashed), or which have not launched as scheduled.  This is an important feature in a system which has defined SLA’s.

While there are a number of passive monitors available (Nagios, etc), there are times where the passive monitor will report the application or service as running, but the app/service is actually doing nothing.  By writing active flight status reporting in the application code itself, the confidence level is higher.  Think of it as an aircraft on autopilot.  Even if the pilot has passed out at the controls on autopilot, the plane will look fine on radar for a while (passive monitoring).  Only direct communication from the pilot via the radio will ensure confidence that the flight is going as intended.

* * * * *

There are a slew of other issues that need to be addressed in design, but these items are the core of protecting your sleep (and sometimes, even your sanity).  These 5 core principles all establish a standard, broad-based view of  a system that keeps everyday operation as simple as possible–and keeps the developer focused on developing.

 

21 Jul 2013
Church Brew Works in Pittsburgh

On a recent business trip to Pittsburgh, I experienced something that really surprised me. I went out to dinner with some co-workers to a microbrewery/restaurant known as Church Brew Works. The name is very appropriate. It is in an old Catholic church building (St Mark’s) which is on the city’s registry as a historical landmark. There is a nice plaque on the outside of the building about its history.

It is a unique concept for me to open a restaurant, let alone a brewery, in what was once a place of worship. But I tend to think towards the utilitarian side, and I became really curious how they remodeled the inside of an old church to make this happen. I also home brew my own beer, so the engineer in me also wanted to see the brewery equipment.

As I entered the restaurant I saw, at the back of the building where the altar was, a large set of brewing equipment for making the beer (pictures below). It emitted a glossy shine from the stainless steel tanks, almost like a mesmerizing aura. And it is, dare I say it… a glorious site. It’s the scale of brewing that a home brewer sometimes dreams about doing.

And then, I catch myself and hear my thoughts wandering to things like “this must be idolatry to be amazed at the site of beer brewing equipment on an altar,” “how could this happen to a place of worship,” and other mental monsters. It was only a defensive response. Any building is only a place of worship when it is actively maintained and honored as such by the group who uses it. This church building, for whatever reason, was abandoned. It’s place in Pittsburgh history preserved by its protection as a recognized landmark, but its time as a place of worship is over.

So I go in to the restaurant with my coworkers and we are led to our seats by our waitress. We proceed to have a very good dinner, and drink some of the beers made here in the building. I had a good time, and I like the place. The beer I chose off the on-tap menu (a Czech-style Pilsner) was very well made.  The stained glass is still in the building which maintains the buildings original character, as well as the original wood and marble architecture, large columns, etc. All are quite beautiful. The place is very open with a lot of space, and there is a lot of echoing (reverberation) when people talk which actually adds to the social atmosphere.

In some ways, it reminded me of dining in a museum cafeteria during a visit, but with the cafeteria being right in the museum instead of a separate food court.  It’s OK, as long as nobody stains the Rembrandt painting.

If you are in the city of Pittsburgh in the future, I highly encourage you to stop by and check it out. Ultimately, it may just be a great (tongue-in-cheek) way of manifesting something we often quote on the internet.

 

 

 

18 Jul 2013
You know you are a Floridian when…

I found this while cleaning out some folders on my hard drive.  After 8 years in the Orlando area, these I can vouch for. The original list is longer, and I removed the items which don’t apply to non-native Floridians.  The ones I’ve marked with asterisks I have found to be particularly true.

=============================

“Down South” means Miami

“Panhandling” means going to Pensacola.

You think no one over 70 should be allowed to drive.

Flip-flops are everyday wear.

Shoes are for business meetings and church.

No, wait, flip flops are good for church too, unless it’s Easter or Christmas.

Sweet tea can be served at any meal.

** You smirk when a game show’s “Grand Prize” is a trip or cruise to Florida.

You measure distance in minutes.

You have a drawer full of bathing suits…. and one sweatshirt.

** You get annoyed at the tourists who feed seagulls.

All the local festivals are named after a fruit.

** A mountain is any hill 100 feet above sea level.

You think everyone from a bigger city has a northern accent.

** You know the four seasons really are: almost summer, summer, not summer but really hot and Christmas.

It’s not soda, cola, or pop…its coke, regardless of brand or flavor, “What kinda coke you want?”

** Anything under 95 is just warm.

** Anything under 70 is chilly.

** You pass on the right and honk at the elderly.

** You understand the futility of exterminating cockroaches.

You can pronounce Okeechobee, Kissimmee and Withlacoochee.

You understand why it’s better to have a friend with a boat than have a boat yourself.

You’ve worn shorts and used the A/C on Christmas.

Socks are only for bowling

You never use an umbrella because you know the rain will be over in five minutes

** A good parking place has nothing to do with distance from the store but everything to do with shade.

You can tell the difference between fire ant bites and mosquito bites

** You know that no other grocery store can compare to Publix

You know that anything under a Category 3 just isn’t worth waking up for.

** You dread lovebug season.

You are on a first name basis with the Hurricane list. They aren’t Hurricane Charley, Hurricane Frances…but Charley, Frances, Ivan and Jeanne.

You know what a snowbird is and you hate them.

You think a six-foot alligator is actually pretty average.

 

 

14 Jul 2013
To Florida, or not to Florida: a quality of life question … maybe

I’ve been really amused over the last few months with some discussions about the best (or at least better) place to live. Currently it’s focused on the city of Pittsburgh or the Central Florida area. The debate between Florida and the rest of the country always seems to be a more heated, emotionally-intense debate.  It may have to do with some subconscious rivalry between the snowy north, and the almost year-round summer in the south–with the occasional tropical storm risk in Florida thrown in as the wild card.

I see these debates differently now than I did a decade or two ago.  I have some thoughts and insight I want to share, with the hope it will help you see these type of debates from a different perspective.

The late American humorist Erma Bombeck wrote a book entitled “The grass is always greener over the septic tank.” Ironically, when we get into debates about what location has the best combination of being the better place to work, live, socialize and play, we are hoping to discover that greener pasture for the next phase of our life.  And I am sure the debates are also to uncover how much manure we are going to have to put up with for that greener pasture, which everyone understandably wants to minimize.

Over the course of my post-High school life, I have lived in eight cities.  Clermont Florida, the most recent, has been a really good experience.  It’s one which I am very thankful for, but it’s also the one which I could not understand why it was so good for me.  I had visited Florida on vacation before, but did not expect it to be so enjoyable and a great fit for me as a place to live.

And, strangely, I began to understand why during a recent trip to Europe with my family. We visited the Austrian town of Kaunertal at the end of skiing season. The town is in a valley surrounded by the Alps, with the skiing on a slope at a 10,500 foot elevation. At the ski shop in the town, I spent a few minutes talking with the young man who was helping us with the rental equipment. He was born and raised in the town, and is an avid snowboarder. And ski season was ending in the Alps in less than a week.

So I asked him, out of curiosity, what he was planning to do in the off-season. The ski shop does close down for the summer, after all. His answer surprised me a little: he was going to go to Chile for 5 months. I asked him why because, to me, it seemed like an odd choice. He pointed out that the Andes mountains in Chile where he goes have some of the best snowboarding in the world. And he takes this trip every year.

And then I considered something I had not thought too much about before: imprinting. For the part of the world he grew up in, winter sports are awesome.  So being able to travel to (or better, escape to) another winter sports location while your hometown is warming up continues the fun.  And when it starts warming up in that other location , go home again–a perpetual winter.  He had grown up in winter, and it was in his blood.

I spent the latter part of my youth in Los Angeles, in the desert heat.  I didn’t have snowy winters, just a little cooler weather than summer.  So after having lived there and in Texas, I was very used to (and enjoyed) warm climates.  For the next 20 years, I spent time in colder climates, and then I moved to Florida.. and my imprinting as a teenager kicked back in.  No wonder I love Florida life so much.

So the next time you hear a debate about Florida or some other area of the country being better, there’s way more to that debate than just the facts.  Even the debaters have their reasons for their positive or negative reactions to different areas of the country–from their own imprinting.